Group testing

In combinatorial mathematics, group testing is a set of problems with the objective of reducing the cost of identifying certain elements of a set.

Contents

Background

Robert Dorfman's paper in 1943 introduced the field of (Combinatorial) Group Testing. The motivation arose during the Second World War when the United States Public Health Service and the Selective service embarked upon a large scale project. The objective was to weed out all syphilitic men called up for induction. However, syphilis testing back then was expensive and testing every soldier individually would have been very cost heavy and inefficient. A basic breakdown of a test is:

Say we have n soldiers, then this method of testing leads to n tests. If we have 70-75% of the people infected then the method of individual testing would be reasonable. Our goal however, is to achieve effective testing in the more likely scenario where it does not make sense to test 100,000 people to get (say) 10 positives.

The feasibility of a more effective testing scheme hinges on the following property. We can combine blood samples and test a combined sample together to check if at least one soldier has syphilis.

Formalization of the problem

We now formalize the group testing problem abstractly.

Input: The total number of soldiers n, an upper bound on the number of infected soldiers d. The (unknown) information about which soldier is infected described as a vector \mathbf{x}= (x_1, x_2, ..., x_n) where x_i = 1 if the item i is infected else x_i = 0.

The Hamming Weight of \mathbf{x} is defined as the number of 1's in x. Hence, |x| \leq d where x is the Hamming weight. The vector \mathbf{x} is an implicit input since we do not know the positions of 1's in the input. The only way to find out is to run the tests.

Formal notion of a Test

A query/test S is a subset of [n]. The answer to the query S \subseteq [n] is defined as follows:

A(S) = \begin{cases} 1, \mbox{ if } \displaystyle\sum_{k\in S} x_k \geq 1\\ 0, \mbox{ otherwise.} \end{cases}

Note that the addition operation used by the summation is the logical-OR, i.e.

A(S) = \displaystyle\bigvee_{i\in S} x_i.

Goal

Compute \mathbf{x} and minimize the number of tests required to determine \mathbf{x}

The question boils down to one of Combinatorial Searching. Combinatorial searching in general can be explained as follows: Say you have a set of n variables and each of these can take on m possible values. So, finding possible solutions that match a certain constraint is a problem of combinatorial searching. The major problem with such questions is that the solution can grow exponentially in the size of the input. Here, we have no direct questions or answers. Any piece of information can only be obtained using an indirect query.

Definition

t(d,n): Given a set of n items with d defects, the minimum number of tests that one would have to make to detect all the defective items is defined as t(d,n).

Consider the case when only one person in the group will test positive. Then if we tested in the naive way, in the best case we would at least have to test the first person to find out if he/she is infected. However, in the worst case one might have to end up testing the entire group and only the last person we test will turn out to really be the one who was infected. Hence, 1 \leq t(d, n) \leq n

Testing Methods

There are two basic principles via which the testing may be carried out:

  1. Adaptive Group Testing is where we test a given subset of items and, get the answer from the of test. We then base the next test on the outcome of the current test.
  2. Non-adaptive Group Testing on the otherhand is when all the tests to be performed are decided a priori

Definition

t^a(d,n): Given a set of n items with d defects, t^a(d,n): is defined as the number of adaptive tests that one would have to make to detect all the defective items.

One should note that in the case of group testing for the Syphilis problem, non-adaptive group testing is crucial. This is because the soldiers might in spread out geographically and adaptive group testing will need a lot of co-ordination.

Mathematical representation of the set of non-adaptive tests

For, S \subseteq [n], define \chi_i \in \{ 0,1 \}^n such that i \in S \Leftrightarrow \chi_s(i) = 1. M is a t \times n matrix of \chi_i. \mathbf{x} is the input vector transposed and \mathbf{x} is the resultant. The construction is based on the grounds that for non-adaptive testing with t tests is represented by a t-subset S_i \subseteq [n] (1 \leq i \leq t). \chi_i for a given i is the i^{th} test. M test matrix where m_{i,j} is one if for the i^{th} test, j \in S. Note that here multiplication is logical AND (\bigwedge) and is logical OR (\bigvee). Then, M \times \mathbf{x} = \mathbf{r} where \mathbf{r} is the resultant of the matrix multiplication. To think of this in terms of testing, it is helpful to visualize matrix multiplication. Here, \mathbf{r} will have a 1 in position i if and only if there was a 1 in that position in both M and \mathbf{x} i.e. if that person was tested with that particular group and if he tested out to be positive. M = \begin{pmatrix} m_{1,1} \cdots m_{1,n} \\ \vdots \ddots \vdots \\ m_{t,1} \cdots m_{t,n} \end{pmatrix}

\mathbf{x} = \begin{pmatrix} x_1 \\ \vdots \\ x_n \end{pmatrix} \mathbf{r} = \begin{pmatrix} r_1 \\ \vdots \\ r_t \end{pmatrix}

Bounds for testing on t^a(d,n) and t(d,n)

1 \leq t^a(d,n) \leq t(d,n) \leq n

The reason for t^a(d,n) \leq t(d,n) is due to the fact that any non-adaptive test can be performed by an adaptive test by running all of of the tests in the first step of the adaptive test. Adaptive tests can be more efficient than non-adaptive tests since the test can be changed after certain things are discovered.

Lower bound on t^a(d,n)

Fix a valid group testing scheme with t tests. Now, for two distinct vectors \mathbf{x} and \mathbf{y} where |\mathbf{x}|, |\mathbf{y}| \leq d, the resulting vectors will not be the same i.e. \mathbf{r(x)} \neq \mathbf{r(y)}. Here \mathbf{r(x)} is the resultant vector when \mathbf{x}. This is because, two valid inputs will never give us the same result. If this ever happened, then we would always have an error in finding both \mathbf{x} and \mathbf{y}. This gives us that the total number of distinct results is the volume of a Hamming Ball of radius d, centered about n i.e. Vol_2(d,n). However, for t bits, the total number of possible distinct vectors is 2^t. Hence, 2^t \geq Vol_2(d,n). Taking the \log on both sides gives us t \geq \log\{Vol_2(d,n)\}.

Now, Vol_2(d,n) \geq {n \choose d} \geq (\frac{n}{d})^d. Therefore, we will end up having to perform a minimum of d\log{\frac{n}{d}} tests.

Thus we have proved, t^a(d,n) \geq d\log\frac{n}{d}

Upper bound on t^a(d,n)

t^a(d,n) \leq O(d\log{n}) .

Since we know that the upper bound on the number of positives is d, we run a binary search at most d times or until there are no more values to be found. To simplify the problem we try to give a testing sccheme that uses O(\log{n}) adaptive tests to figure out a i such that x_i = 1. The related problem is solved by splitting [n] in two halves and querying to find a 1 in one of those and then proceeding recursively to find the exact position in the half where the query returned a 1. This will take 2\lceil\log{n}\rceil time or if the first query is performed on the whole set, it will take \lceil\log{n}\rceil %2B1. Once a 1 is found, the search is then repeated after removing the i^{th} co-ordinate. This can be done at most d times. This justifies the running time of O(d\log{n}) . For a full proof and an algorithm for the problem referto: CSE545 at the University at Buffalo

Upper bound on t(1,n)

t(1,n) \leq \lceil\log{n}\rceil This upper bound is for the special case where d = 1 i.e. there is a maximum of 1 positive. In this case, the matrix multiplication gets simplified and the resultant \mathbf{r} represents the binary representation of i for test i. This gives a lower bound of \lceil\log{n}\rceil. Note that decoding becomes trivial because the binary representation of i gives us the location directly. The group test matrix here is just the parity check matrix H_m for the [2^m - 1, 2^m-m-1, 3] Hamming code.

Thus as the upper and lower bounds are the same, we have a tight bound for t(d,n) when d = 1. Such tight bounds are not known for general d.

Upper Bounds for Non-Adaptive Group Testing

For non-adaptive group testing upper bounds we shift focus toward disjunct matrices. Disjunct matrices have been used for many of the bounds because of their nice properties. Through use of different constructions of disjunct matrices it has been shown that \Omega(\frac{d^2}{\log{d}}\log{n}) \leq t(d,n). Also for upper bounds we currently have that (i) t(d,n) \leq \mathcal{O}(d^2 \log{n}) (explicit construction) and (ii) t(d,n) \leq \mathcal{O}(d^2 \log^2{n}) (strongly explicit construction). It is good to note that the current known lower bound for t(d,n) is already a \frac{d}{\log{d}} factor larger than the upper bound for t^a(d,n). Another thing to note is that give the smallest upper bound and biggest lower bound they are only off by a factor of \frac{1}{\log{d}} which is fairly small.

See Also

References

  1. Atri Rudra's course on Error Correcting Codes: Combinatorics, Algorithms, and Applications (Spring 2007), Lectures 7.
  2. Atri Rudra's course on Error Correcting Codes: Combinatorics, Algorithms, and Applications (Spring 2010), Lectures 10, 11, 28, 29
  3. Dorfman, R. The Detection of Defective Members of Large Populations. The Annals of Mathematical Statistics, 14(4), 436-440. Retrieved from [1]
  4. Du, D., & Hwang, F. (2006). Pooling Designs and Nonadaptive Group Testing. Boston: Twayne Publishers.
  5. Ely Porat, Amir Rothschild: Explicit Non-adaptive Combinatorial Group Testing Schemes. ICALP (1) 2008: 748-759